Modeling decision making processes

ABSTRACT

A computer implemented method of predicting decisions uses the knowledge of one or more individuals. The individuals, referred to as a team, are knowledgeable about the domain in which decisions are being made. The team individually rates the importance of decision criteria they deem relevant. They then rate the extent which multiple problem characteristics are deemed relevant to the decision. The ratings are subjected to automated quantitative analysis for consistency, and the raters may discuss and modify inconsistent ratings if appropriate. Once the ratings are accepted, the raters then rate the decision options against the highest scoring problem characteristics as determined in the initial ratings. After one or more further rounds of consistency evaluations, the highest rated options are selected as the prediction of the decision to made by the adversary.

FIELD OF THE INVENTION

The present invention relates to decision making, and in particular tomodeling decision making processes to predict actions by others whetheror not they participate in the process.

BACKGROUND OF THE INVENTION

Prior attempts at determining what people will decide when presentedwith a decision to make involve combinations of intuition, individualbiases or probabilistic assessments. Individuals or groups may considerthe multiple factors related to the decision to be made in trying todetermine what decision will be made. In the case of groups, voting onthe likely decision may be used to arrive at an answer. There is a needfor a method to provide a more accurate assessment of the likelydecision to be made.

Such a method may be used to identify potential likely actions bycriminals and terrorist groups, as well as identifying other types ofdecisions.

SUMMARY OF THE INVENTION

A method of predicting decisions uses the knowledge of one or moreindividuals. The individuals, referred to as a team, are knowledgeableabout the domain in which decisions are being made. The teamindividually rates the importance of each decision criterion they deemrelevant. They then rate the extent which multiple problemcharacteristics are deemed relevant to each decision criterion. Theratings are subjected to quantitative analysis to assess theirconsistency and to highlight differences of opinion, and the raters maydiscuss and modify inconsistent ratings if appropriate. Once the ratingsare accepted, the raters then rate the extent to which each of the knowndecision options fulfills the highest scoring decision criteria asdetermined in the initial ratings. After one or more further rounds ofconsistency evaluations, and subsequent discussion, the highest ratedoptions are selected as the best prediction by this team of the decisionto be made. The method permits a range of varied opinions to be entered,weighted, and automatically combined to obtain a consensus prediction ofthe decision to be made. The method treats variability in the opinionsof raters not as noise but as useful information in making predictions,and avoids individual biases or probabilistic assessments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart of a process for predicting decisions of others inone example embodiment.

FIG. 2 is a simplified block diagram of a computer system for executingat least portions of the process of FIG. 1.

FIG. 3 is a depiction of the characteristics of a complex domainrequiring the protection of critical infrastructure and the predictionof attack scenarios.

FIG. 4 is a top view of a potential target identifying specific threats.

FIG. 5 is a block diagram of a decision process for predicting adecision.

FIG. 6 is a block diagram of a process for identifying defenses.

FIG. 7 is a chart illustrating an analysis of the extent to whichvarious decision criteria associated with the goals of the decisionmaker are met by various outcome characteristics.

FIG. 8 is a chart illustrating an analysis of the extent to whichvarious decision criteria associated with the capabilities of thedecision maker are met by various outcome characteristics.

FIG. 9 is a chart illustrating the use of covariation analysis to createratings scales for factors associated with the decisions.

FIG. 10 is an example plot illustrating prioritization of threat domainsbased on difficulty of logistics versus size of the impact.

FIG. 11 illustrates the use of an Ishikawa diagram to support scenariodevelopment for attacking specific targets.

FIG. 12 is a Pareto chart illustrating overall threat of attack for atarget based on relative risk, impact and logistics cost.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, reference is made to the accompanyingdrawings that form a part hereof, and in which is shown by way ofillustration specific embodiments in which the invention may bepracticed. These embodiments are described in sufficient detail toenable those skilled in the art to practice the invention, and it is tobe understood that other embodiments may be used and that structural,logical and electrical changes may be made without departing from thescope of the present invention. The following description is, therefore,not to be taken in a limited sense, and the scope of the presentinvention is defined by the appended claims.

The functions or algorithms described herein can be implemented as a setof procedures, or as computer software, or as a combination of softwareand human implemented procedures in one embodiment. If used, thesoftware comprises computer executable instructions stored on computerreadable media such as memory or other type of storage devices. Multiplefunctions are performed in one or more modules as desired, and theembodiments described are merely examples. The software is executed on adigital signal processor, ASIC, microprocessor, or other type ofprocessor operating on a computer system, such as a personal computer,server or other computer system.

The present invention imposes a rigorous, documented, and principledprocess in group discussions involving the prediction of decisions andactions. Whereas existing decision making tools such as decisioncomponent weighting and quality function deployment have been used tohelp a group make a decision, this process is used to guide the group topredict the actions of others by systematically considering all of thepotentially relevant decision criteria and outcome characteristics.Further, this process does not require group consensus, and insteadpermits group judgments to be derived from weighted combinations of thebeliefs of the members of the group.

In one embodiment, a series of interconnected spreadsheets are used toguide the entry of a range of varied opinions, which are individuallyassessed and then automatically combined to obtain a consensusprediction of a decision to be made. An example of such a decisioninvolves predicting which targets are most likely to be attacked bydifferent attacking each with their own goals and means generally at100. Examples of spreadsheets and data presentations used in the processare shown in following figures.

A team of individuals knowledgeable about the decision maker and thedecision domain (“experts”) is recruited at 105. Diversity in theopinions of these experts is valued. In this example, such a team wouldhave some knowledge of issues that would be considered by the attackers,such as their goals and the means available to them to carry out thosegoals.

The team is instructed to list the decision criteria—the issues thatwill be considered by the person(s) making the decision—at 110. Thesemay be listed as the title of the rows of a matrix on paper or in aspreadsheet. In the present example, the issues could include a list ofthe goals of the attack (e.g., news coverage, ransom paid), and theoperations concerns of the attacker (e.g., financial resourcesavailable, the need for the attackers to successfully escape).

At 115, the team is instructed to determine the relative importance ofthese criteria on a scale from 1 to 10 in one example. Other scales maybe used as desired. If there is general agreement, the weights aresimply listed in an importance column. If there is disagreement, theaverage of the weights is used. If it is agreed that the attacker caresmore about the success of the attack than the escape of the attackers,for example, attack success will be rated relatively higher, and captureavoidance relatively lower. These ratings may be entered as the secondcolumn in each of the relevant rows of a matrix on paper or in aspreadsheet.

The team is then instructed at 120 to identify characteristics of thedecision outcome(s) that may be related to the decision criteria. Forthe current example, these characteristics might include the presence ofsecurity systems at the target site, the site's location adjacent tointerstate highways that can be used as an escape route, and the numberof people required to attack the target site. These are entered as thetitle of the columns of a matrix on paper or in a spreadsheet. For eachof these characteristics, the experts are asked to state how they wouldbe measured, which is added to each description. The units used forrepresenting the measure for each characteristic may be varied asdesired. For instance, closeness to escape routes may be measured bydriving time in seconds or minutes, etc.

At 125, the team rates the degree to which each of these outcomecharacteristics is related to a decision criteria, using a 0, 1, 3, 9rating scale in one embodiment. Each team member produces a rating foreach combination of decision criterion and outcome characteristic. Thus,a team member may decide that escape route proximity relates very highlyto capture avoidance, but not at all to the amount of ransom paid.

At 130, an analysis of the covariation of the judgments of the teammembers is completed. For example, in a spreadsheet embodiment, a suiteof statistics is calculated and then highlighted on the fly. Thevariation in expert ratings is reflected in the standard deviation oftheir ratings for a particular combination of characteristic andcriterion. The agreement of experts with each other is reflected in theintercorrelation matrix of rater judgments across criteria and outcomecharacteristics. These statistics are computed and the variation ishighlighted with a color code (green, yellow, red) or other attribute.The experts' average ratings for each combination of characteristic andcriterion are computed and placed in the appropriate column. Theseratings are then multiplied by the weights determined in 115 todetermine scores and overall ranking of each of the decisioncharacteristics, both for individuals and for the team as a whole.Ratings similarly calculated for each rater are compared to those of theteam as a whole. Finally a concordance analysis is carried out todetermine the extent to which the rankings of the team as a whole aredifferent from those of the individuals.

These analyses describe the variability in expert judgments and thesource of that variability. Three sources of variability in particularare highlighted: errors (e.g., mistakes in generating the ratings),systematic differences of opinion in individual raters, and systematicdifferences of opinion among groups of raters. Mistakes are quickly andeasily corrected, ensuring that following analyses are valid.Differences of opinion may be based upon a lack of mutual understandingof the definitions of the decision criteria, or differences in beliefabout the salience of those decision criteria in the current context.

The team is instructed to analyze the quantitative data and makeappropriate adjustments at 135. Two kinds of adjustments in particularare of interest. Individual experts may, after listening to discussion,determine that their ratings are different from the teams' because theexpert's basis for making the rating was inconsistent with those ofother ratings. Corrections of this type lead to improved consistency.Alternatively, difference in ratings may reflect real differences inopinion about what will matter in the actual decision process, oruncertainty about what that process entails. These differences, oncevalidated, improve the diversity of coverage of the decision space andare retained. In particular, ratings among subgroups of experts thatdiffer from the ratings of other subgroups of experts represent “schoolsof thought” that, once discovered, can be analyzed for their underlyingassumptions and evidentiary support. Once the analysis is complete, anassessment of the relative importance of each characteristic (e.g., asdepicted in a scree plot) is used to determine how many decisioncharacteristics to carry forward to the next stage of the process.

At 140, the validated subset of decision characteristics, and theirassociated weights, are entered as the titles of rows of a secondmatrix, which may be embodied in a second spreadsheet. These are treatedhenceforth as the weighted decision criteria for selecting amongindividual options (possible decision outcomes).

The preceding steps result in a decision making model that reflects theraters' mutual understanding of the decision criteria, weightings, andcharacteristics, and the underlying structure of any differences ofopinion that exist among participating experts. This initial processresults in a list of decision criteria that is robust, validated, andeasy to use.

The experts then generate a list of decision options which is entered at145 as the titles of columns of the second matrix. For example, a set ofspecific crime scenarios can be listed. Typically the preceding analysisresults in a relatively small list of potential choices, the processdoes not require it. All of the scenarios for attacking all of themilitary installations in a region can be entered, if need be. Themethods for generating decision options depend on the domain understudy, but could include various cause-and-effect analysis tools such asIshikawa tools, Failure Mode and Effects Analysis (FMEA), or Managementof Change tools.

The raters rate the extent to which each of these decision optionsfulfills the decision criteria using a 0, 1, 3, 9 rating scale at 150.Thus, an attack on an equipment depot may be rated highly on closenessto escape routes and less highly on the impact of the attack on publicopinion.

As in 130, a suite of statistics is calculated at 155, and is reviewedby the rating team at 157.

The resulting ordered list of options 160 is the process' prediction ofthe most likely outcome of the decision. Further, the difference inscores of each option provides an index of the probability of theprediction. If an attack on a military depot has a much higher scorethan any other alternative, then the tool is indicating that this is themost likely decision from the list that has been subjected to analysis.

A block diagram of a computer system that may be used to execute atleast portions of the above algorithm, such as covariance analysis onexpert judgments in matrices, is shown in FIG. 2. Any type ofspreadsheet-based application may be used. The functions may also beprogrammed directly into a stand-alone application if desired. A generalcomputing device in the form of a computer 210, may include a processingunit 202, memory 204, removable storage 212, and non-removable storage214. Memory 204 may include volatile memory 206 and non-volatile memory208. Computer 210 may include—or have access to a computing environmentthat includes—a variety of computer-readable media, such as volatilememory 206 and non-volatile memory 208, removable storage 212 andnon-removable storage 214. Computer storage includes RAM, ROM, EPROM &EEPROM, flash memory or other memory technologies, CD ROM, DigitalVersatile Disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium capable of storing computer-readable instructions.Computer 210 may include or have access to a computing environment thatincludes input 216, output 218, and a communication connection 220. Thecomputer may operate in a networked environment using a communicationconnection to connect to one or more remote computers. The remotecomputer may include a personal computer, server, router, network PC, apeer device or other common network node, or the like. The communicationconnection may include a Local Area Network (LAN), a Wide Area Network(WAN) or other networks.

Computer-readable instructions stored on a computer-readable medium areexecutable by the processing unit 202 of the computer 210. A hard drive,CD-ROM, and RAM are some examples of articles including acomputer-readable medium. For example, a computer program 225 capable ofproviding a generic technique to perform access control check for dataaccess and/or for doing an operation on one of the servers in a COMbased system according to the teachings of the present invention may beincluded on a CD-ROM and loaded from the CD-ROM to a hard drive. Thecomputer-readable instructions allow computer system 200 to providegeneric access controls in a COM based computer network system havingmultiple users and servers.

FIG. 3 is a block diagram of a more complex domain illustrating scopingof the problems associated with protecting critical infrastructure fromattacks. Those charged with defending critical infrastructure may tendto focus on the infrastructure they own or control, or upon scenariosthey personally believe see are particularly risky, or uponinstallations they believe are poorly protected. However, the protectiontask can not be systematically addressed based on such individualjudgments. There are thousands of potential targets 300, and hundreds ofvulnerabilities per target. Such targets include buildings, malls,airports, power grids, dams and bridges, capitols, sports venues,synagogues, and water works to name a few. It is simply not feasible toeliminate every vulnerability at every potential target.

The consequences 330 of attacks on these targets should also beconsidered. Such consequences include blackouts, economic losses,contaminated cities, refinery disabled, mass casualties, airportshutdown and no drinking water to name a few. These potentialconsequences are not likely to be fully understood by the defenders, norequally valued by an adversary.

There are also many different attacks or exploits 310 that can becarried out against each of the targets' vulnerabilities. Commonexploits include cyber attack, staged accidents, explosives, biologicalagents, armed attacks, chemical agents and radioactivity. While thereare protective measures or safeguards 320 that may be implemented, suchas access control, guards, physical security, explosive detection,x-rays/scanners, control rooms, cyber security, emergency responseteams, closed circuit TV and redundant systems, these all take resourcesto implement and none of them is comprehensively effective. It isapparent that resources to provide safeguards against all possibleexploits for all possible targets are simply not available.

FIG. 4 is an example of a potential target identifying specific attacks,each of which may have multiple methods. Given a target, such as anairport 400, there are several features most likely to be involved in anattack. Such features include parking garages 410, electric substations420, perimeter gate 430, off airport threats 440, aviation fuel storage450, departing aircraft 460 and tunnels under runways 470, not tomention the main terminal 480. It is difficult to determine whichfeature is most likely to be attacked.

FIG. 5 is a block diagram of a decision process instantiated for anairport scenario in accordance with the method of FIG. 1 at 500. Aformal, iterative method is implemented for combining the assessments ofmultiple experts to assess risk. The experts use the potentialattackers' perspective. The method permits revalidation and reassessmentas data arrive, and enables identification of countermeasures, even forunpredicted attack scenarios.

The adversary may not themselves understand the details of theirdecision process; it has to be estimated based on available data. Thisleads to an approach that start with first principles—what is themotivation?—and only then gets to intermediate goals and executionstrategies. The process is iterative, because the experts leverage andrefine each other's understanding. Even with a lack of consensus on thebig picture, there can be consensus on important threads such asspecific vulnerabilities and defenses. Differences of opinion will beidentified and quantified, leading to either discussion and resolution,or the generation of alternative scenarios. The high level block diagram500 shows multiple elements that are shown in further detail infollowing figures. First, a goals, means and methods analysis 510,corresponding to blocks 105 through 125, is performed, followed by afactor analysis 520, corresponding to 130-135. This leads to a domainassessment 530 (135-140), scenario development 540 (145), riskassessment 550 (150-155), risk analysis 560 (157-160) and technologydevelopment 570 to reduce the risk. 570 is representative of the resultsof the process. The elements are arranged in a circle with arrowsbetween the elements to represent the iterative nature of thevulnerability function assessment.

FIG. 6 is a block diagram 600 showing further detail of selected blocksof FIG. 5. A systematic analysis is performed at each of multiplestages, each building on the preceding analysis. The method incorporateslogistics concerns as well as the impact of the goals of the actions.Bias of the target's owners is reduced.

Impact features and impact goals are considered in the team identifyingcritical features 610. Logistic features and issues are also consideredat 620. 610 and 620 correspond to 510. At 630 (520), the identifiedcritical impact features and critical logistics features are used toidentify critical domains at 640 (530) to identify criticalinfrastructure domains at risk. Critical scenarios and requirements forthe scenarios are generated for multiple critical infrastructures at 650(540). Vulnerabilities for each scenario 660 (550) are then identifiedat 670 (560) along with responses at 680 (570), identifying the bestdefenses.

FIG. 7 is a chart 700 illustrating the systematic assessment of impactof an action on the potential targets. It provides further detailcorresponding to 610/510. The analysis is sensitive to tangible andintangible goals. The chart is an illustration of a spreadsheet thatprovides several operations goals in a first column 710, such asdestabilization of the US economy, inhibit US ability to wage war,destabilize US political system, etc. Multiple columns 720 are thenprovided for assessing the impact of particular results of actions, suchas killing civilians, killing military personnel, damaging things thatare expensive to repair, etc. A row 730 then provides a relativeimportance measurement for each such impact, and the impacts are rankedon relative importance at 740. A correlation of the rater to eachrelative importance measurement is then provided at 750 for feedback tothe rater.

FIG. 8 is a chart 800 illustrating logistics associated with operationsissues. It provides further detail corresponding to 620/510. Importantoperations issues are shown in a column 810, and include issues such asthe cost of the operation, the risk of discover and probability ofsuccess. At row 820, several logistics are presented that are related tothe operations issues, such as amount training/preparation required,level of communication and coordination required and need to meetdifficult timing constraints. The team members, or raters rank eachthese at 825, and a relative importance measurement is calculated at830. A rank on relative importance is then provided at 840, and acorrelation of the rater to the item average is provided at 850.

FIG. 9 is a chart 900 illustrating the creation of ratings scales forfactors associated with the decisions. It provides further detailcorresponding to 630/520. In one embodiment a cluster analysis, or otheranalysis is used to remove duplicative scales and collapse the analysisto core issues. In effect, the most important factors are identified,redundancy is reduced, and measurable criteria are developed by thisanalysis.

FIG. 10 is a plot 1000 illustrating prioritization of threat domainsbased on difficulty of logistics versus size of the impact. It providesfurther detail corresponding to 640/530. To get to this point, thescales are used to rate the domains on those scales. Domains are ratedby worst-case scenarios. Results are likely to change through aniteration or two. Surprises are likely here, and represent the successof the process in reflecting the aims of the attacker instead of our ownexpectations. In one example, religious gatherings came out with a highimpact, with a low difficulty of logistics, while a military base had ahigh difficulty of logistics, yet a relatively low impact.

FIG. 11 is an Ishikawa diagram illustrating scenario development inattacking a target such as an airport. It provides more detailcorresponding to 650/540. It uses simulated data for illustrationpurposes only. The diagram presents a systematic method for developingscenarios, driven by cause-effect analysis, but also includes impactattributes (e.g., stealthy vs. spectacular attacks).

FIG. 12 is a Pareto chart 1200 illustrating overall threat of attack fora target based on relative risk, impact and logistics cost. It providesmore detail corresponding to 660/550. The goal is not so much toperfectly predict every attack (although clearly the devastatingscenarios need to be responded to), but to identify commonvulnerabilities to efficiently utilize resources in protecting suchvulnerabilities.

1. A method using a team of individual raters to generate a decisionmaking model for predicting decisions, the method comprising:identifying possible motivations of a decision maker; entering a varietyof opinions about a strength of such motivations; weighting themotivations; combining the weights to create a decision making model;identifying possible decision outcomes; and assessing the possibledecision outcomes with respect to the decision making model.
 2. Themethod of claim 1 and further comprising: generating a list of decisionoptions; the raters rating the extent to which each of these decisionoptions meets their opinions; calculating a suite of statistics forreview by the team; generating an ordered list of options as aprediction of the most likely outcome of the decision process.
 3. Themethod of claim 2 wherein differences of opinion on each option providesan index of the uncertainty of the prediction.
 4. The method of claim 3and further comprising incorporating logistics factors.
 5. A computerimplemented method using a team to generate a decision making model forpredicting decisions, the method comprising: identifying issues likelyto be considered in making a decision in a decision domain; determiningrelative importance of the identified issues; identifyingcharacteristics of issues related to making a decision; individuallyrating the degree to which the characteristics are related to making thedecision; determining rankings of individuals and team identifiedcharacteristics; and iteratively adjusting individual ratings based onthe rankings to generate the decision making model.
 6. A method ofpredicting a decision in a decision domain by another party, the methodcomprising: recruiting a team of individual raters knowledgeable aboutthe decision domain; the team listing decision criteria that may beconsidered by the another party; listing outcome characteristics; theteam rating the relevance of the outcome characteristics to eachdecision criteria; assessing a covariance in ratings using a statisticalanalysis; selecting highly rated outcome characteristics for use in adecision model; generating a list of decision outcomes based on highestrated outcome characteristics; each team member rating the extent towhich each decision outcome addresses the outcome characteristics;assessing a covariation in judgments using statistical analysis toproduce a weighted list of options corresponding to predictions of thedecision.
 7. The method of claim 6 and further comprising: identifyingissues likely to be considered in making a decision in a decisiondomain; determining relative importance of the identified issues;identifying characteristics of issues related to making a decision;individually rating the degree to which the characteristics are relatedto making the decision; determining rankings of individuals and teamidentified characteristics; and adjusting individual ratings based onthe rankings to generate the decision making model.
 8. The method ofclaim 7 and further comprising: generating a list of decision options;the raters rating the extent to which each of these decision optionsmeets the decision criteria; calculating a suite of statistics forreview by the team; generating an ordered list of options as aprediction of the most likely outcome of the decision process.
 9. Themethod of claim 7 wherein difference in scores of each option providesan index of the uncertainty of the prediction.
 10. The method of claim 9and further comprising incorporating logistics factors.
 11. The methodof claim 6 and further comprising adjusting individual ratings ofoutcome characteristics based on the covariation analysis of suchoutcome characteristics.
 12. The method of claim 6 and furthercomprising adjusting individual ratings of decision options based on thecovariation analysis of such decision options.
 13. The method of claim 6and further comprising generating a weighted list of options as aprediction of the decision outcome.
 14. A computer assisted method usinga team to generate a decision making model for predicting decisions, themethod comprising: identifying issues likely to be considered in makinga decision in a decision domain; determining relative importance of theidentified issues; identifying characteristics of issues related tomaking a decision; individually rating the degree to which thecharacteristics are related to making the decision; determining rankingsof individuals and team identified characteristics; and iterativelyadjusting individual ratings based on the rankings to generate thedecision making model.
 15. A physical computer readable medium havinginstructions for causing a computer to implement a method using a teamof individual raters to generate a decision making model for predictingdecisions, the computer implemented method comprising: recordingpossible motivations of a decision maker identified by the team ofindividual raters; recording a variety of opinions about a strength ofsuch motivations; weighting the motivations; combining the weights tocreate a decision making model stored in memory accessible by thecomputer; recording possible decision outcomes identified by the team ofindividual raters; and creating a list of the possible decision outcomeswith respect to the decision making model and indication of ranking ofthe possible decision outcomes fixed for recording on physical media foruse in determining one or more most likely decisions.
 16. A physicalcomputer readable medium having instructions for causing a computer toimplement a method using a team of individual raters to generate adecision making model for predicting a most likely target, the computerimplemented method comprising: recording possible motivations of adecision maker identified by the team of individual raters; recording avariety of opinions about a strength of such motivations; weighting themotivations; combining the weights to create a decision making modelstored in memory accessible by the computer; recording possible decisionoutcomes identified by the team of individual raters; and creating alist of the possible decision outcomes with respect to the decisionmaking model and indication of ranking of the possible decision outcomesfixed for recording on physical media for use in determining one or moremost likely targets, enabling security assets to be efficientlyutilized.
 17. The computer readable medium of claim 16 wherein thedecision making model comprises a plurality of weighted matricesrepresenting motivations and decision outcomes.
 18. A computer assistedmethod of predicting decisions, the method comprising: using a team suchthat members on the team list decision criteria that are recorded onmemory in the computer; rating importance of each decision criterion byeach member and recording the rating on memory in the computer;generating a list of outcome characteristics for the decisions; eachmember rating a relevance of each outcome characteristic to eachdecision criterion and recording such relevance on memory in thecomputer; assessing a covariation of outcome characteristics usingcomputer implemented statistical analysis; adjusting member ratings foroutcome characteristics as a function of the statistical analysis;selecting highly rated outcome characteristics; creating and storing adecision model from such selected highly rated outcome characteristics;generating a list of decision outcomes from such highest rated outcomecharacteristics; each member rating an extent to which each decisionoutcome addresses such outcome characteristics and recording suchrating; assessing covariation in such ratings using computer implementedstatistical analysis; optionally adjusting member ratings as a functionof such covariation analysis; and generating a weighted list ofpredicted decisions using the computer adapted for display, transmissionor storage.